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PLAN  FOR  THE  EVALUATION  OF  JOB  ASSISTANCE  CENTERS 


Introduction 

Job  Assistance  Centers  (JAC)  provide  assistance  to  Army  service  personnel, 
civilian  employees,  and  their  family  members  who  are  transitioning  from  military  jobs  to 
employment  in  the  civilian  sector.  The  services  provided  by  the  JAC  are  designed  to 
prqMure  individuals  for  a  successful  job  search.  The  centers  are  run  under  the  auspices 
of  the  Army  Career  and  Alumni  Program  (ACAP),  which  is  part  of  the  Department  of 
Defense  Transition  Assistance  Program.  The  National  Defense  Authorization  Act  of 
1991  requires  that  the  Dqnrtments  of  Defense,  Labor,  and  Veterans  Affairs  provide  job 
transition  assistance  to  military  and  civilian  personnel  who  would  lose  their  jobs  due  to 
installation  closures  and  the  downsizing  and  consolidation  of  military  units.  The  JAC 
program  is  the  Army’s  response  to  that  mandate. 

The  Dqnrtroent  of  Labor  (DoL),  also  in  respcmse  to  the  Congressional  mandate, 
has  established  Transition  Assistance  Offices  (TAO)  near  DoD  installations  nationwide. 
The  JAC  and  TAO  often  serve  the  same  Army  and  other  DoD  installations.  The 
programs  of  assistance  provided  by  the  JAC  and  TAO,  however,  are  designed  to 
complement  each  other.  The  £)oL  program  provides  group  seminars  or  classes  designed 
to  increase  the  general  job  search  skills  of  the  separating  service  personnel.  The  JAC,  on 
the  other  hand,  {Htwide  more  individualized  counseling  services  geared  to  the  particular 
vocational  experience  and  plans  of  the  transitioning  soldiers,  Army  civilians,  and  their 
family  members.  Among  the  services  offered  by  the  JAC  are  the  Army  Employers  and 
Alumni  N^ork  (AEAN),  the  Defense  Outplacement  Referral  System  (DORS),  the 
Transition  Bulletin  Board  (TBB),  the  Federal  Job  Information  Center  (FJIC)  files,  help 
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in  prqnring  resumes  and  cover  letters,  and  counseling  on  job  interviewing  techniques, 
salary  n^otiations,  and  evaluating  job  offers. 

Questims  have  been  raised,  however,  about  the  cost  effectiveness  of  maintaining 
the  JAC  program  at  locations  where  the  DoL  program  is  also  available  (the  DoL 
program  is  not  available  overseas).  Specifically,  the  added  value  of  the  JAC  program  to 
transitioning  Army  personnel  and  their  family  members  has  been  questioned  by  the  DoD 
Inspector  Genml.  Essentially,  this  evaluation  of  the  JAC  is  being  conducted  to 
determine  the  answer  to  three  basic  questions: 

How  do  JAC  clients  compare  with  non-JAC  clients  in  regard  to  post-separation 
employment? 

Witch  JAC  services  have  the  most  impact  on  success  in  achieving  employment? 

Considering  the  additional  costs  o/proinding  JAC  services,  are  the  added  benefits 
derived  Jrom  the  program  cost  effective? 

The  first  question  can  be  answered  in  a  number  of  ways,  depending  upon  the 
aq)ect  of  employm«it/unemployment  examined.  Table  1  presents  13  types  of  program 
criterion  or  outcome  mea«ires  that  could  be  obtained  in  the  separate  samples.  These 
measures,  along  with  others  that  could  be  defined,  would  constitute  the  dependent 
variables  in  the  planned  multivariate  analyses  (see  section  on  Analysis).  The  analyses 
would  focus  on  differences  in  post-separation  employment  of  recipients  and  non- 
recipients  of  JAC  services  while  controlling  for  personal  and  family  factors  and  the 
military  job  experience  of  the  sampled  individuals. 

The  answer  to  the  second  question.  Which  JAC  services  have  the  most  impact  on 
success  in  achieving  employment,  would  also  be  answered  through  multivariate  analyses. 
Whether  or  not  individuals  received  given  JAC  services  can  be  ascertained  from  the  JAC 
data  base  and  confirmed  in  follow-up  interviews.  Variables  measuring  receipt/nonreceipt 
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Table  1.  Types  of  Program  Outcome  or  Criterion  Measures 


•  Time  unemployed  since  separation 

•  Time  between  separation  and  first  job  search 

•  Percent  of  separatees  drawing  unemployment  compensation 

•  Number  of  weeks  on  unemployment  compensation 

•  Percent  of  separatees  with  full-time  employment 

•  Annualized  salary  of  full-time  workers 

•  Match  between  position  desired  and  position  obtained 

•  Consistency  of  job  with  career  goals 

•  Evaluation  of  job  transition  services  received 

•  Helpfulness  of  transition  services  in  coping  with  civilian  life 

•  Satisfaction  with  transition  services  received 

•  Goodwill  value  of  transition  services  (toward  encouraging  others  to 
join  Services) 

•  Entrance  into  Reserves/National  Guard 


of  particular  services  can  be  created.  The  effects  (direct,  indirect,  and  total)  of  these 
measures  on  the  outcome  measures  can  then  be  ascertained  within  the  context  of 
model(s)  that  attempt  to  capture  the  interrelationships  among  all  the  variables  of 
interest. 

The  third  question  is  perhaps  the  most  difficult  to  answer.  Assuming  that  the 
multivariate  analyses  indicate  that  individuals  who  received  JAC  services  are  more 
successful  in  obtaining  post-separation  employment,  the  employment  differential  or 
program  benefits  obtained  must  be  evaluated  in  terms  of  the  monetary  costs  of  providing 
the  job  assistance  services.  The  approach  planned  for  evaluating  the  cost  effectiveness  of 
the  JAC  .services  is  also  described  in  the  Analysis  section. 

The  remainder  of  this  report  focuses  on  the  major  methodological  aspects  of  the 
planned  evaluation.  As  the  administration  of  a  questionnaire  will  be  required  to  collect 
the  data  needed  for  the  evaluation,  the  first  major  task  of  the  evaluation  team  will  be  to 
develop  a  ques^’-^nnaire.  At  the  same  time  the  questionnaire  is  being  developed,  an 
approach  to  sampling  recipients  and  nonrecipients  of  JAC  services  should  be  articulated. 
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Both  the  questionnaire  and  sampling  approach  should  be  developed  in  a  timely  fashion 
to  enable  a  clearance  package  to  be  prepared  and  submitted  to  the  Office  of 
Management  and  Budget  (OMB)  soon  after  the  initiation  of  the  evaluation.  The  actual 
collection  of  the  data  and  an  approach  to  the  analyses  of  the  data  are  the  next  major 
methodological  aspects  of  the  evaluation  described  in  this  plan.  The  last  section  of  this 
plan  briefly  describes  the  two  reports  that  should  be  produced  by  the  evaluation  team. 

Questionnaire  Development 

A  single  questionnaire  with  appropriate  branch  instructions  for  different  types  of 
respondents  would  probably  be  the  easiest  to  develop,  administer,  and  analyze.  This 
would  facilitate  obtaining  a  fairly  common  set  of  data  for  most  respondents.  The 
questionnaire  should  build  a«  much  as  possible  on  questionnaires  that  have  already  been 
developed  for  use  with  JAC  service  recipients,  e.g.,  the  ACAP  Follow-up,  Solider  Exit, 
and  Army  Employer  and  Alumni  Network  (AEAN)  Questionnaires.  Available  data 
concerning  response  distributions  on  the  items  of  these  earlier  questionnaires  should  be 
examined  before  adopting  the  items  for  use  in  the  current  evaluation  survey.  The 
questionnaire  length  should  be  restricted  to  ensure  that  on  the  average  approximately  20 
minutes  will  be  sufficient  to  collect  the  required  information  from  the  respondents. 

The  first  draft  of  the  questionnaire  should  be  examined  very  carefully  by  the 
organization  that  will  be  responsible  for  its  administration  as  well  as  by  the  COR  and 
ACAP  monitoring  staff.  It  should  be  clear  that  the  questionnaire  will  produce  the 
requisite  evaluative  information  and  that  it  does  not  contain  any  poorly  worded  or 
superfluous  items  or  improper  branching  instructions.  After  appropriate  revisions,  if  any, 
the  questionnaire  should  be  tried  out  on  a  sample  of  nine  soldiers,  their  spouses,  and 
Army  civilians  who  have  received  JAC  job  transition  services.  The  questionnaire  should 
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be  administered  in  person  so  that  any  problems  that  the  respondents  had  with  the 
questions  will  be  more  apparent  and  can  be  explored  more  fully.  Again,  after 
appropriate  revisions  if  any,  the  questionnaire  should  be  administered  via  computer 
assisted  telephone  interviews  to  a  sample  of  nine  respondents.  The  respondents  should 
be  representative  of  the  nine  different  subsamples  proposed  (see  the  Sampling  Approach 
Section). 

Besides  examining  the  tryout  data  for  problems  with  individual  items,  the  length  of 
time  the  respondents  took  to  answer  each  question  and  the  questionnaire  as  a  whole 
should  be  carefully  examined.  If  necessary,  further  revisions  to  the  questionnaire  should 
be  made  to  ensure  that  the  questionnaire  can  be  administered  in  20  minutes  on  the 
average. 

The  completed  questionnaire,  along  with  a  statement  attesting  to  its  approval  by 
an  Institutional  Review  Board  for  Protecting  Human  Research  Subjects,  a  sampling  plan, 
a  data  collection  plan,  and  an  overview  of  the  analysis  should  be  packaged  for  submission 
for  OMB  clearance.  The  draft  questionnaire  should  be  completed  in  the  first  month  of 
the  evaluation  effort,  the  final  questionnaire  and  OMB  approval  package  by  the  end  of 
the  second  month. 

Sampling  Approach 

Proposed  Subsamples 

Nine  separate  random  subsamples  are  proposed.  Eight  of  the  subsamples  would 
consist  of  300  cases  each,  while  the  ninth  subsample  would  consist  of  600  cases.  All 
sampled  ex-military  and  Army  civilian  personnel  will  have  left  their  military  related 
service  between  6  and  18  months  prior  to  their  selection  into  the  samples.  (The  samples 


5 


should  not  be  drawn  until  the  0MB  clearance  process  is  almost  completed.)  The  nine 
subsamples  are  listed  below. 


1.  A  subsample  of  300  Marines.  These  Marines  will  consist  of  personnel  who 
have  been  through  the  DoL  program  and  personnel  who  have  not. 

Whether  they  have  been  through  the  program  will  be  determined  after  they 
are  contacted.  Marines  who  happened  to  use  Army  JAC  services  in 
addition  to  or  in  lieu  of  DoL  services  would  not  be  excluded  from  the 
sample. 

2.  A  subsample  of  300  Air  Force  exservicemembers.  Like  the  Marines,  their 
participation  in  the  DoL  program  and  possibly  the  JAC  will  be  determined 
after  they  are  contacted. 

3.  A  similar  random  subsample  of  300  Navy  exservicemembers. 

4.  A  subsample  of  300  Army  civilian  ex-employees  who  utilized  JAC  program 
services  (according  to  JAC  records). 

5.  A  subsample  of  300  Army  civilian  ex-employees  who  did  not  utilize  JAC 
program  services. 

6.  A  subsample  of  300  spouses  of  Army  exservicemembers  who  utilized  JAC 
program  services  (according  to  JAC  records). 

7.  A  subsample  of  300  spouses  of  Army  exservicemembers  who  did  not  utilize 
JAC  program  services.  These  spouses  would  be  selected  from  the  spouses 
of  the  Army  exservicemembers  who  constitute  subsamples  8  and  9  below. 

(It  is  anticipated  that  over  50%  of  the  sampled  Army  exservicemembers 
will  be  married  and  that  only  a  small  proportion  of  their  spouses  will  have 
utilized  JAC  program  services). 
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8. 


A  subsample  of  300  Army  exservicemembers  who  did  not  utilize  JAC 
program  services  (according  to  JAC  records). 

9.  A  subsample  of  600  Army  exservicemembers  who  did  utilize  JAC  program 
services  and  most  likely,  TAO  services.  (This  sample  is  larger  than  the 
others  in  order  to  allow  more  accurate  assessment  of  the  impact  of  the 
separate  JAC  service  elements  on  the  outcome  measures  for  Army 
exservicemembers.) 

The  nine  snbsamples  as  defined  above  allow  comparisons  between  Army 
exservicemembers,  their  spouses,  and  civilian  ex-employees  who  utilized  JAC  services  and 
who  did  not  utilize  those  services.  It  allows  comr)arisons  of  these  groups,  especially  the 
Army  exservicemembers,  with  exservicemembers  from  the  other  Services  who  most  likely 
used  only  TAO  services,  if  they  used  any  job  transition  services.  Moreover,  the  size  of 
the  subsamples  will  permit  fairly  stable  subpopulation  estimations  to  be  made. 

The  size  of  the  subsamples  will  also  allow  the  utilization  in  multivariate  analyses  of 
dichotomous  (1/0)  dummy  variables  that  measure  whether  the  individual  case  is  in  a 
particular  group  or  not  (e.g.,  whether  the  individual  is  an  exMarine  or  not).  The 
following  initial  set  of  dummy  variables  are  proposed: 

(1)  Was  in  the  Marines 

(2)  Was  in  the  Air  Force 

(3)  Was  in  the  Navy 

(4)  Was  in  the  Army 

(5)  Was  an  Army  civilian  employee 

(6)  Is  the  spouse  of  a  soldier 

(7)  Received  JAC  services 
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As  the  proposed  multivariate  analyses  proceed  (see  Analysis  section),  one  or  more  of 
these  dichotomous  variables*  could  be  replaced  by  more  comprehensive  or  more  finely 
defined  variables.  For  example,  the  first  three  variables  could  be  combined  into  a  single 
variable.  Was  in  the  Service,  not  Army;  the  seventh  variable  could  be  dropped  in  favor  of 
a  set  of  variables  indicating  which  JAC  services  had  been  received. 

The  sampling  plan  should  be  completed  by  the  end  of  the  second  month  of  the 
evaluation  effort. 

Service  Member  Sampling 

The  Defense  Manpower  Data  Center  (DMDC)  Active  Duty  Military  Loss  Files 
(ADMLF)  and  the  DoD  Separation  Address  Files  (SAF)  can  be  used  to  draw  random 
samples  of  Army,  Navy,  Air  Force,  and  Marine  personnel.  Data  are  stored  in  the 
ADMLF  on  a  quarterly  basis  by  fiscal  year.  Among  the  personal  items  of  information 
stored  are  SSN,  name,  sex,  date  of  birth,  education  level,  AFQT  percentile  score,  marital 
status,  number  of  dependents,  and  home  of  record.  The  military-related  items  of 
information  include  date  of  separation,  months  of  service  at  separation,  primary  and  duty 
occupation,  pay  grade,  and  time  in  grade  at  separation,  and  reenlistment  eligibility  code 
for  enlisted  personnel. 

The  SAF  covers  a  similar  population  to  that  of  the  ADMLF.  In  addition  it 
provides  a  mailing  address  for  each  separatee.  Omitted  from  the  file  are  personnel  who 
separated  due  to  death,  retirement,  or  reenlistment,  personnel  who  have  re-joined  the 
active  or  reserve  military  service;  and  personnel  who  are  not  eligible  for  reenlistment.  As 


'  Care  must  be  exercised  in  the  utilization  of  these  dummy  variables  to  avoid  having  any  one  variable  in 
an  analysis  completely  determined  by  the  remaining  dummy  variables. 
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the  mailing  addresses  in  the  SAF  may  not  be  current,  the  Equifax  National  Change  of 
Address  File  can  be  used  to  obtain  more  up-to-date  addresses  for  separatees  who  have 
recently  moved. 

Am^  Qvflian  Sampling 

The  Army  Civilian  Personnel  System  (ACPERS),  which  is  maintained  by  the 
Civilian  Personnel  Systems  Directorate,  can  be  used  to  extract  a  random  sample  of  Army 
civilian  employees  who  recently  left  the  Army.  The  date  the  employee  left  Army 
employment  (whether  voluntarily,  through  a  Reduction-in-Force  [RIF],  or  to  retire)  can 
be  used  to  help  establish  the  sampling  window.  Once  the  sample  has  been  selected 
(through  use  of  the  last  SSN  digits),  the  ACPERS  files  can  be  used  to  obtain  employee 
addresses,  specific  and  major  occupation  codes,  date  of  birth,  pay  grade/level, 
race/national  origin,  sex,  and  infonnation  about  training,  special  skills,  education  and 
tenure. 

JAC  User  Sampling 

The  JAC  maintains  a  data  base  containing  the  name  SSN,  MOS,  and  other 
information  including  types  of  transition  services  received  for  all  recipients  of  the 
program  services.  This  data  base  can  be  used  to  obtain  the  requisite  random  samples  of 
soldiers,  spouses,  and  Army  civilians  who  received  JAC  services  between  6  and  18 
months  prior  to  their  selection  in  the  samples.  By  comparing  the  JAC  data  file  with  the 
DMDC  and  ACPERS  files,  a  determination  can  be  made  of  which  soldiers  and  Army 
civilians  did  not  use  the  JAC  prior  to  transitioning  out  of  their  Army  jobs.  Comparison 
of  the  JAC  files  with  the  other  files  will  also  allow  determination  of  the  sizes  of  the 
subpopulations  involved. 
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Sampling  Weights 

Tlie  sampling  weights  for  the  individuals  in  the  sampled  groups  will  differ  as  a 
function  of  the  ratio  of  the  population  and  sample  sizes  in  the  strata  h-om  which  they  are 
drawn.  These  original  sampling  weights  should  be  adjusted  for  non-responses  or  missing 
respondents.  If  some  extreme  sampling  weights  are  produced  in  a  given  stratum  by  the 
non-response  adjustments,  the  largest  and  smallest  weights  should  be  trimmed  ( replaced 
by  the  largest  and  smallest  remaining  non-extreme  weights)  in  order  to  reduce  the 
variance  of  the  survey  population  estimates. 

The  subsamples  should  be  drawn  when  final  OMB  clearance  is  imminent.  The 
sampling  weights  should  be  derived  when  the  data  collection  is  completed  (approximately 
four  months  after  receipt  of  OMB  clearance). 

Data  Collection 

Data  Collection  Facilities 

Data  not  available  on  the  automated  DMDC,  ACPERS,  and  JAC  files  will  be 
collected  through  the  use  of  the  developed  questionnaire  administered  by  computer 
assisted  telephone  interviews  (CATI).  The  telephone  research  centers  from  which  the 
calls  are  made  should  be  located  in  different  time  zones  to  facilitate  contacting 
respondents  during  "prime  time"  (3:00  p.m.  to  8:00  p.m.).  The  centers  should  use  a 
common  CATI  system  and  share  the  same  schedules,  database,  and  computing  facilities. 

To  facilitate  dialing  selected  telephone  numbers,  the  CATI  system  should  contain 
an  auto-dialer.  Call-backs  should  be  handled  by  a  scheduling  system  that  is  updated  on¬ 
line  when  calls  do/do  not  go  through.  If  a  busy  telephone  is  reached,  a  callback  should 
be  automatically  rescheduled  for  a  later  time.  Call-backs  should  be  as  extensive  as 
required  to  achieve  an  overall  response  rate  of  75  percent. 
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The  current  interviewer  instructions,  questions,  and  response  options  should  be 
presented  on  an  automated  screen,  with  questions  clearly  distinguished  from  instructions 
and  coding  options.  When  multiple  questions  are  put  on  a  single  screen,  each  question 
should  have  a  separate  input  6eld  associated  with  it  Interviewers  should  be  able  to  enter 
responses  directly  onto  the  screen  and  into  the  data  base.  Every  keystroke  entered 
through  the  keyboard  should  also  be  entered  into  an  audit  trail  file  which  stores  the  data 
in  case  the  interview  is  prematurely  terminated  and  later  restarted.  The  audit  trail  file 
will  also  prevent  data  loss  in  the  event  of  computer  failure. 

Out-of-range  responses  and  inconsistent  or  implausible  responses  (based  on 
responses  to  previous  questions)  should  be  identified  on-line  so  that  the  interviewer  can 
make  further  queries  during  the  interview.  The  CATI  system  should  allow  on-line 
correction  of  information  received  and,  if  necessary,  rebranching  to  a  corrected  path. 

The  CATI  system  should  also  allow  non-intrusive  interview  supervision  and 
observation  of  conversations.  During  the  first  week(s)  of  monitoring,  the  interviewers 
should  be  given  immediate  feedback  on  any  incorrect  techniques  used  in  gaining 
cooperation,  asking  questions,  or  recording  responses.  Common  problems  that  arise 
across  interviewers  should  be  identified  and  further  training  given  if  warranted. 
Interviewer  Training 

The  interviewees  should  at  all  times  be  aware  that  they  are  representing  the  U.S. 
Army  in  the  conduct  of  an  important  survey.  Although  none  of  the  questions  asked  will 
be  of  a  highly  personal  nature,  some  of  the  questions  may  touch  on  sensitive  areas  for 
particular  respondents  (e.g.,  post-separation  employment  questions  for  unemployed  ex- 
servicemembers).  Gaining  rapport  with  the  respondents  and  assuring  them  that  their 
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responses  are  completely  confidential  will  be  vital  to  obtaining  complete  and  accurate 
data.  Only  highly  skilled  interviewers  should  be  used. 

During  training,  the  interviewers  should  be  taught  to  distinguish  readily  between 
instructions  and  questions  to  be  read  to  the  respondent.  The  interviewers  should  be 
thoroughly  familiar  with  the  questionnaire  items  and  the  branching  instructions.  The 
interviewers  should  know  the  difference  between  hard  range  edits  that  set  absolute  upper 
and  lower  limits  of  allowable  responses,  and  soft  range  checks  that  should  trigcer  queries 
concerning  implausible  (but  not  impossible)  responses.  They  should  also  be  ughly 
familiar  with  the  procedures  for  reporting  calls  ending  in  refusal  o.  in  some  problem 
status  so  that  there  will  be  a  permanent  record  of  circumstances  surrounding  the  call  and 
valuable  contact  related  tips  in  case  another  interviewer  is  assigned  to  call  back  the 
respondent. 

Data  collection  should  be  completed  within  four  months  after  the  receipt  of  OMB 
clearance  for  the  questionnaire. 

Analyses 

A  three-phase  multivariate  analysis  is  planned  to  answer  the  first  two  evaluation 
questions  (see  page  2).  The  question  answers  would  essentially  be  found  in  the  statistical 
significance  of  variables  representing  JAC/nonJAC  clients  and  the  various  JAC  services  in 
equations  capturing  the  interrelationships  of  these  variables  with  measures  of 
servicemembers’  personal  characteristics,  service  employment  experience,  and  post¬ 
separation  employment.  The  approach  planned  to  answer  the  third  basic  evaluation 
question,  Are  the  benefits  derived  from  the  JAC  program  cost  effective,  is  discussed  after  the 
description  of  the  planned  multivariate  analyses. 
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Multivariate  Analyses 

The  first  phase  of  the  multivariate  analysis  will  concentrate  on  the  selection  of 
variables  within  the  framework  of  a  relatively  simplified  overall  model  (see  Figure  1). 

The  analyses  would  be  conducted  on  60%  of  the  servicemember  sample  and  would 
involve  the  full  set  of  initial  variables.  Hierarchical  multiple  regression  analyses  would  be 
used  to  identity  the  variables  that  account  for  substantial  proportions  of  the  program 
outcome  measures  (see  Table  1).  In  the  second  phase,  which  would  also  be  conducted 
on  the  60%  servicemember  subsample,  the  reduced  set  of  variables  would  be 
incorporated  into  one  or  more  LISREL  (Linear  Structural  RELations)  models  and  the 
degree  to  which  the  model(s)  fit  the  data  assessed.  If  necessary,  changes  in  the  model(s) 
will  be  made  to  improve  the  fit.  In  the  third  phase,  the  final  model(s)  will  be  tested 
using  the  40%  servicemember  subsample.  In  addition,  direct  group  comparisons  will  be 
made  to  supplement  the  modelling  results.  Each  of  these  analytic  phases  is  described  in 
more  detail  below. 

Phase  I.  Figure  1  presents  the  initial  model  that  would  be  used  in  the  process  of 
reducing  the  number  of  variables  and  formulating  less  simplified  models  to  explain  the 
variance  of  the  outcome  measures  across  setvicemembers.  The  proposed  initial  causal 
model  hypothesizes  direct  effects  or  linkages  between  Personal  and  Family 
Characteristics  (e.g.,  gender,  AFQT  percentile,  age,  marital  status,  number  of 
dependents)  and  the  outcome  measures.  Personal  and  Family  Characteristics  are  also 
hypothesized  to  have  indirect  effects  on  the  Outcome  Measures  through  their  impact  on 
Service  Employment  Experience  (e.g.,  rank,  military  occupational  skills,  time  in  service, 
location  when  separated,  type  of  separation)  and  Job  Assistance  Services  provided  or 
used  (e.g.,  AEAN,  DORS,  TBB,  FJIC,  and  Resume  Writer).  Service  Employment 
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Job  Assstanoe 
Sofvioss  Rootivecl 
(AEAN.OORS.TBB. 
Indtvk^  CounMling) 


Personal  &  Family 
Characteristics 
(Gender,  AFOT 
Percentiles,  Race, 
Marital  Status,  No.  of 
Dependents) 


Output  Measures 
(Time  Unemployed, 
Annualized  Salary, 
Evaluation  of  Job 
Transition  Services) 


Service  Emptopent 
Expenence  (f^, 
Job  Cornplexity, 
Yesrsof^rvioe) 


Figure  1.  Initial  Simplified  Model 


Experience  variables,  in  turn,  are  hypothesized  to  have  a  direct  impact  on  the  outcome 
measures  as  well  as  an  indirect  effect  on  the  Outcome  Measures  through  their  impact  on 
the  Job  Assistance  Services  used.  Finally,  the  model  hypothesizes  that  the  Job  Assistance 
Services  provided  directly  impact  the  Outcome  Measures.  Note  that  the  model  does  not 
specify  any  causal  relationships  among  variables  within  the  sets  of  variables,  comprising 
the  measures  of  Personal  and  Family  Characteristics,  Service  Employment  Experience, 
and  the  Job  Assistance  Services.  (Such  causal  relationships  may  be  hypothesized  in  later 
models). 

The  analyses  of  the  initial  model  will  be  conducted  through  hierarchical  analyses 
of  variable  sets.  The  variables  that  are  considered  exogenous  in  the  model  (the  Personal 
and  Family  Characteristics)  would  be  inputed  into  a  multiple  regression  equation  first, 
followed  by  the  Service  Employment  History  variables,  and  the  measures  of  the  Job 
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Assistance  Services  provided.  By  examining  and  statistically  testing  the  resultant  increase 
in  after  each  variable  set  is  introduced,  one  can  determine  the  added  explanatory 
power  of  each  variable  set.  By  examining  the  significance  levels  and  changes  in  the 
equation  regression  coefficients,  one  can  determine  the  total,  direct,  and  indirect  effects 
of  the  individual  variables  on  the  outcome  measure  or  dependent  variable  (see  Cohen 
and  Cohen,  1983). 

The  model  would  be  run  separately  for  each  of  the  outcome  measures  (see 
Table  1).  Variables  entering  into  the  equations  significantly  will  be  identified  and 
incorporated  into  later  less  simplified  models.  The  rationale  of  each  of  the  outcome 
measures  as  well  as  the  intercorrelations  among  the  outcome  measures  will  also  be 
examined  to  determine  whether  some  outcome  measures,  e.g.,  evaluation  of  job 
transition  services  received,  should  be  considered  in  the  model(s)  to  be  causally  related 
to  other  outcome  measures,  e.g.,  time  unemployed  since  separation. 

Perhaps  the  most  difficult  measures  to  obtain  in  applying  the  model(s)  will  be 
those  that  describe  the  military  (and  civilian)  jobs  of  the  sampled  servicemembers. 
Military  occupation  is  obviously  an  important  factor  to  keep  in  mind  when  evaluating  the 
transition  from  the  military  to  the  civilian  sector.  For  example,  Mangum  and  Ball  (1989) 
found  greater  transfer  of  training  for  technical  military  jobs  than  general  military  training 
or  combat  type  assignments.  Thus,  if  occupation  is  not  measured  and  controlled  for 
there  is  the  possibility  of  erroneously  concluding  that  one  transition  program  is  less 
effective  than  another.  Fortunately,  there  are  a  number  of  job  taxonomies  that  permit 
comparisons  across  military  service  jobs. 

The  Department  of  Labor’s  Dictionary  of  Occupational  Titles  (DOT)  together 
with  the  results  of  the  military  Occupational  Crosscode  Project  enable  not  only  cross- 
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service  comparisons  at  the  job  level  but  also  permit  groupings  on  the  basis  of  various  job 
characteristics.  DoD  has  equated  military  (enlisted  and  officer)  and  civilian  jobs  on  the 
basis  of  job  analytic  criteria  including  tasks  performed,  machines  and  tools  used,  and 
output  and  results  achieved  (the  DMDC  crosscode  and  Crosswalk  Files).  Once  military 
jobs  are  assigned  a  DOT  code,  they  can  be  grouped  according  to  occupation  types  or  on 
more  abstract  or  psychological  dimensions  such  as  complexity.  For  example,  using  Fine’s 
worker  function  ratings  contained  in  the  DOT,  complexity  groups  could  be  formed.  The 
DOT  also  contains  ratings  regarding  training  time,  aptitudes,  temperaments  and  interests, 
and  physical  demands  and  environmental  conditions.  Cross-service  job  comparisons 
could  be  made  on  the  basis  of  the  cluster  they  fall  into  on  the  basis  of  their  factor  scores 
on  the  above  types  of  DOT  job  ratings.  Finally,  should  the  linkage  to  the  DOT  be 
unwarranted,  cross-service  job  comparisons  could  precede  on  the  basis  of  the  DoD 
Occupational  Conversion  Manual  (1991).  Enlisted  jobs  can  be  categorized  into  10  areas, 
68  groups,  and  160  subgroups.  Officer  jobs  can  be  categorized  into  9  occupational  areas 
and  66  occupational  groups.  It  is  possible  to  aggregate  jobs  to  an  even  higher  level,  if 
necessary,  by  combining  occupational  areas  into  broader  categories  such  as  combat  arms, 
technical,  and  other.  However,  such  a  high  level  of  aggregation  may  cause  one  to 
overlook  potentially  important  job  dimensions. 

The  same  kinds  of  variables  used  to  measure  the  military  job  experience  of  the 
servicemember  can  be  used  to  measure  the  kind  of  job  the  servicemember  has  either 
obtained  or  is  aspiring  to  obtain.  A  similarity  index  or  distance  function  could  be  used  to 
assess  the  congruence  of  the  service  and  civilian  occupations. 

Phase  II.  After  selecting  the  variables,  one  or  more  structural  equation  models 
will  be  constructed  that  reflect  the  evaluation  team’s  notions  about  the  variable 
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interrelationships.  The  model(s)  will  imply  a  particular  pattern  of  correlations  among  the 
variables.  Comparison  of  the  model's  correlation  matrix  with  the  sample  correlation 
matrix  will  give  an  indication  of  how  well  the  model  accounts  for  the  data--the  greater 
the  correspondence,  the  better  the  model  explains  (i.e.,  "fits")  the  data.  Such  an 
assessment  of  model  fit  can  be  conducted  using  the  LISREL  software  package  (version  7; 
Joreskog  &  Sorbom,  1989).  Although  the  coefficients  for  the  path  model  could  be 
estimated  using  traditional  path  analytic  methods,  these  methods  do  not  provide  indexes 
of  fit  based  upon  the  observed  and  fitted  correlation  matrices.^  Indexes  of  fit  include 
the  goodness-of-fit  index  which  generally  ranges  between  zero  and  one  (although  negative 
values  are  possible),  larger  values  being  associated  with  good  models;  and  a  chi-square 
statistic,  which  is  better  described  as  a  "badness-of-fit"  measure-if  significant,  it  means 
there  are  significant  differences  between  the  model’s  estimated  correlation  matrix  and  the 
observed  sample  correlation  matrix. 

In  addition  to  estimates  of  the  model’s  path  coefficients  (the  direct  effects), 
indirect  and  total  effects,  coefficients  of  determination  (both  for  variables  and  the  system 
of  equations),  fit  statistics,  and  modification  indexes  wiU  be  requested.  These  values  are 
given  for  each  parameter  that  has  not  been  estimated  in  the  model  (i.e.,  for  each  path 
coefficient  equal  to  zero).  The  modification  index  represents  the  minimum  reduction  in 
the  chi-square  statistic  that  is  expected  if  the  fixed  parameter  in  question  is  freed  (i.e., 
estimated).  Large  modification  indexes  suggest  relationships  in  the  data  the  current 
model  does  not  explain.  Modifications  should  be  made  to  the  model  only  if  justified  on 
theoretical  grounds.  In  addition,  the  altered  models  should  be  fit  to  data  from  a  new 

^  See  Mulaik,  James,  Van  Alstine,  Bennett,  Lind,  &  Stillwell,  1989,  for  an  evaluation  of  goudness-of-fii- 
indices  for  LISREL  models. 
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sample  (i.e.,  cross-validated)  to  rule  out  the  possibility  that  the  modifications  were  due  to 
chance  fluctuations  in  the  original  sample  (MacCullum,  1986).  (This  will  be  done  in  the 
next  phase  of  the  analysis.) 

Phase  III.  After  the  third  and  final  LISREL  analysis  on  the  60%  sample,  the 
developed  model(s)  will  be  run  on  the  40%  random  sample.  The  goodness-of-fit  indexes 
obtained  on  this  independent  sample  will  be  compared  with  those  obtained  on  the  60% 
sample.  The  direct,  indirect,  and  total  effects  of  the  variables  on  individual  readiness  in 
both  samples  will  also  be  compared.  The  LISREL  model(s)  will  then  be  run  on  the  total 
sample  to  obtain  more  stable  estimates  of  the  model  variable  effects. 

Analysis  of  variance  will  be  used  to  compare  the  means  of  the  dependent 
variables  of  the  various  groups  of  serviceraembers.  Army  civilians,  and  spouses.  Analyses 
of  covariance  will  also  be  used  to  compare  the  mean  values  of  the  these  groups  when 
controlling  for  differences  in  selected  personal  characteristics  and  job  experience  of  the 
group  members.  The  covariates  used  will  be  variables  appearing  in  the  final  LISREL 
model(s). 

Cost-Benefit  and  Cost-Effectiveness  Analysis 

There  is  no  objective,  scientific  way  of  making  decisions,  nor  is  it  likely  that  there 
ever  will  be.  However,  use  of  the  techniques  developed  by  decision  theory,  cost-benefit, 
and  cost-effectiveness  analysis  can  provide  the  decision  maker  with  a  useful  framework 
and  language  for  describing  and  discussing  the  effectiveness  of  Job  Assistance  Centers  in 
helping  soldiers  and  Department  of  the  Army  civilians  leaving  their  military  jobs  to  locate 
new  jobs.  These  techniques  can  help  clarify  the  existence  of  alternatives,  decision  points, 
gaps  in  information,  and  value  judgments  concerning  trade-offs. 
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Traditional  cost-benefit  analysis  is  an  application  of  economics  that  has  been  used 
to  analyze  various  kinds  of  decisions,  particularly  large  investments  in  public  goods. 
Decision  theory,  although  in  some  ways  closely  related  to  cost-benefit  analysis,  has 
developed  as  a  separate  discipline.  It  differs  from  cost-benefit  analysis  in  its  emphasis  on 
tracing  through  the  consequences  of  each  of  a  series  of  sequential  decisions,  in  its 
substitution  of  the  values  of  the  decision-maker  for  market  prices,  and  in  its  use  of 
probability  theory  to  deal  with  the  problem  of  uncertainty.  For  reasons  explained  below, 
traditional  cost-benefit  analysis  has  not  been  very  useful  in  making  decisions  about 
services  such  as  Job  Assistance  Centers,  The  analytical  tools  of  decision  theory  do 
provide  a  useful  way  of  assembling  or  displaying  information,  but  should  not  be  relied  on 
to  make  the  final  decision.  If  for  no  other  reason,  decision  theory  is  limited  because 
executives  will  simply  not  take  time  to  assign  quantitative  values  to  the  varied  elements 
that  enter  into  any  particular  decision. 

Cost-benefit  analysis  is  not  a  rule  or  formula  which  would  make  the  decision  or 
predetermine  the  choice  for  the  decision-maker.  Rather,  it  refers  to  the  systematic 
analysis  and  evaluation  of  alternative  courses  of  action  drawing  upon  the  analytical  tools 
and  insights  provided  by  economics  and  decision  theory.  It  is  a  framework  and  a  set  of 
procedures  to  help  organize  the  available  information,  display  trade-offs,  and  point  out 
uncertainties.  In  this  way,  cost-benefit  analysis  can  be  a  valuable  aid;  but  it  does  not 
dictate  choices,  nor  does  it  replace  the  ultimate  authority  and  responsibility  of  the 
decision-maker. 

The  following  sections:  (1)  describe  the  limitations  of  a  "cost-benefit"  framework 
for  evaluating  the  costs  and  benefits  of  the  Job  Assistance  Center  services,  (2)  discuss  the 
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ways  in  which  analysis  can  assist  in  making  responsible  decisions,  and  (3)  outline  a  "cost- 
effectiveness"  methodology  for  evaluating  the  Army’s  Job  Assistance  Centers. 

!  imitot  rtf  Cost-Benefit  Analysis 

The  most  important  and  pervasive  limitation  of  cost-benefit  analysis  is  the  role  of 
values  (Campbell,  1988;  Cascio,  1989;  Goldstein,  1991).  Many  of  the  factors  that  are 
likely  to  be  most  significant  in  a  decision  concerning  the  Job  Assistance  Centers  cannot 
be  measured  in  common  terms  (such  as  dollars)  that  are  agreeable  to  all  concerned 
parties.  Different  individuals  place  different  values  on  output  measures  such  as  percent 
full  time  employment,  weeks  of  unemployment  compensation,  and  total  time  unemployed 
after  separation.  Thus,  an  analysis  that  assigns  a  quantitative  value  to  one  or  more  of 
these  factors  is  necessarily  subjective  and,  to  some  degree,  arbitrary. 

The  primary  way  in  which  analysis  can  simplify  decisions  is  by  reducing  the 
relevant  factors  to  numbers  that  can  be  added,  subtracted,  and  compared.  The  only  unit 
generally  considered  feasible  for  doing  this  is  the  dollar,  but  the  use  of  dollar  values 
poses  a  great  many  problems. 

Many  factors  caimot  be  satisfactorily  expiessed  in  dollar  terms  because  they 
involve  important  values  on  which  there  is  no  agreement.  Some  factors,  such  as  time 
unemployed  after  separation  or  percent  full  time  employment,  are  difficult  or  impossible 
to  express  in  dollar  terms,  even  apart  from  the  value  question,  because  they  are  not 
traded  in  the  market ...  or  thought  about  in  monetary  units. 

A  major  difficulty  with  using  market  prices  is  that  the  definition  of  benefits  and 
costs  differ,  depending  on  the  vantage  point  adopted.  The  benefits  and  costs  for  a 
particular  subgroup  may  be  different  from  the  benefits  and  costs  for  the  group 
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considered  as  a  whole.  This  difficulty  aris  -s  from  the  broader  problem  of  the  distribution 
of  benefits  and  costs. 

The  way  benefits  and  costs  are  distributed  through  time  and  across  different 
segments  of  the  population  is  crucial  in  many  program  decisions,  but  quantification, 
conversion  to  dollar  terms,  and  aggregation  often  obscure  important  distributional 
considerations.  It  is  difficult  to  compare  benefits  and  costs  from  multiple  vantage  points. 
The  function  of  the  decision  maker  is  to  weigh  the  impact  of  a  decision  on  numerous 
groups;  so  it  follows  that  even  when  the  benefits  or  costs  can  all  be  stated  in  common 
dollar  units,  they  may  not  be  additive. 

The  distribution  of  benefits  and  costs  over  time  poses  even  greater  difficulties.  In 
traditional  cost-benefit  analysis,  the  value  of  future  benefits  and  costs  is  reduced  by  using 
a  discount  rate.  One  justification  for  using  a  discount  rate  is  that  resources  invested  to 
yield  future  benefits  cannot  be  used  now  for  present  benefits;  thus  the  discount  rate 
reflects  the  opportunity  cost  of  postponing  benefits,  just  as  interest  earned  on  savings 
accounts  compensates  for  the  postponement  of  spending.  The  use  of  discount  rates  has 
been  the  subject  of  much  dispute  in  traditional  cost-benefit  analysis. 

A  characteristic  of  most  decisions  is  that  the  full  consequences  of  the  decision  are 
impossible  to  predict  with  any  certainty  at  the  time  the  decision  is  made.  Also,  the 
economic  data  used  to  analyze  economic  impacts  are  typically  old  (i.e.,  not  representative 
of  the  immediate  situation)  by  the  time  they  are  processed  into  usable  form,  and  their 
reliability  declines  rapidly  thereafter.  The  use  of  uncertain  data  adds  a  significant 
subjective  element  to  any  analysis. 
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Uses  of  Cost-Benefit  Analysis 


The  discussion  of  the  limitations  of  cost-benefit  analysis  is  important  so  as  to 
dispel  exaggerated  ideas  about  what  such  analysis  can  do,  and  to  emphasize  the  types  of 
problems  that  surround  any  attempt  at  formal  analysis  of  costs  and  benefits.  But  it 
should  be  remembered  that  the  relevant  baseline  against  which  analytical  methods  must 
be  judged  is  not  some  ideal  system  of  value-free  mathematics,  but  rather  the  intuitive,  ad 
hoc,  unsystematic,  and  often  frantic  methods  now  used.  Compared  to  the  latter,  methods 
of  cost-benefit  analysis,  using  techniques  and  concepts  adopted  from  economics  and 
decision  theory,  represent  a  significant  advance. 

Cost-Effectiveness  versus  Cost-Benefit  Analysis 

The  cost-effectiveness  criterion  suggests  that  the  job  assistance  program  (i.e..  Job 
Assistance  Centers  or  DoL’s  Transition  Assistance  Program)  chosen  should  be  the  one 
that  meets  the  desired  output  goals  (e.g.,  reduced  unemployment  compensation)  at  the 
lowest  cost.  In  comparing  transition  program  alternatives  (e.g..  Army  vs  DoL)  the 
desired  output  goal  or  measure  is  held  constant.  It  is  assumed  that  this  goal  can  be  met 
by  each  alternative  at  some  cost,  even  if  this  cost  is  very  large.  The  job  transition 
alternatives  are  then  compared  based  on  their  costs. 

The  cost-benefit  formulation  implies  that  both  costs  and  transition  service 
performance  would  vary  among  the  alternatives.  This  is  not  undesirable,  in  principle. 

The  overall  criterion  for  evaluating  alternatives  in  the  cost-benefit  instance  would  be  to 
choose  the  transition  alternative  for  which  the  difference  between  the  value  of  the 
transition  service  performance  produced  by  the  alternative  and  its  cost  are  the  greatest. 

A  particularly  difficult  problem  in  the  analysis  of  job  transition  programs  is 
estimating  the  benefits  of  different  programs  (i.e..  Army  Job  Assistance  Centers)  for 
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increasing  the  ease  of  job  transition  for  soldiers  and  Department  of  the  Army  civilians. 
This  class  of  problem  is  one  that  Harris,  et  al.  (1991)  considered  in  the  context  of 
developing  a  cost-performance  tradeoff  model  for  helping  to  estimate  entry  standards  for 
recruits. 

The  cost-effectiveness  framework  is  a  subset  of  the  more  general  cost-benefit 
analysis.  A  cost-effectiveness  approach  allows  the  analysts  to  circumvent  the  issue  of 
placing  an  explicit  dollar  value  on  the  output  of  the  job  transition  program  (i.e.,  percent 
full  time  employment,  weeks  of  unemployment  compensation,  etc.).  An  overall  criterion 
for  evaluating  transition  programs  within  a  cost-effectiveness  framework  is  to  choose  that 
job  transition  program  which  meets  overall  performance  goals  for  soldiers  and 
Department  of  the  Army  civilians  at  the  lowest  total  cost.  Hence,  the  assumption  is  that 
overall  job  transition  goals  can  be  met,  at  some  cost,  by  all  the  job  transition  program 
alternatives,  and  that  the  alternatives  will  be  compared  based  on  costs.  These  costs 
include  both  fixed  and  variable  job  transition  program  costs.  The  next  section  presents 
an  overview  of  the  basic  process  to  be  followed  in  applying  the  cost-effectiveness 
framework  to  job  transition  program  evaluation.  The  process  is  summarized  in  terms  of 
the  approximate  sequence  of  events  followed  in  carrying  out  a  cost-effectiveness  analysis 
(Sticha  et  al.,  1988;  Sticha  et  al.,  1990). 

OTst-F.ffectiveness  Anatyms 

A  cost-effectiveness  analysis  is  conducted  in  the  phases  shown  in  Figure  2.  Each 
phase  consists  of  a  sequence  of  events  and  activities  carried  out  to  plan  and  conduct  the 
cost-effectiveness  analysis.  The  phases  are  summarized  in  the  sequence  they  occur 
below. 
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Figure  2.  Cost-Effectiveness  Analysis  Phases 

The  first  step  in  a  cost-effectiveness  analysis  is  to  state  the  performance 
objective(s)  to  be  attained  by  the  job  transition  programs  to  be  compared.  Next  is  study 
conceptualization  and  planning.  This  phase  is  carried  out  in  seven  steps  as  follows:  (1) 
delineate  the  job  transition  program  issue  in  question;  (2)  specify  tasks  to  be 
accomplished;  (3)  develop  assumptions  and  constraints;  (4)  determine  usage ‘patterns  and 
job  transition  scenarios  to  specify  where  and  how  job  transition  services  are  to  be 
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employed;  (5)  develop  essential  elements  of  analysis  to  answer  cost-effectiveness  analysis 
objectives;  (6)  determine  alternative  job  transition  programs  to  be  compared  in  the  cost- 
effectiveness  analysis;  and  (7)  develop  task  assigmnents  for  activities  required  to  perform 
the  cost-effectiveness  analysis.  These  steps  provide  inputs  into  portions  of  the  study  plan 
applicable  to  planning  effectiveness  analysis  and  cost  analysis. 

The  effectiveness  analysis  phase  is  performance  assessment-oriented  and  is 
planned  and  conducted  in  close  coordination  and  in  parallel  with  the  cost  analysis  phase. 
The  purpose  of  this  phase  is  to  determine  the  relative  effectiveness  of  the  job  transition 
programs  being  compared  in  terms  of  how  well  each  mediates  attainment  of  desired 
performance  objectives.  This  phase  is  carried  out  in  two  parts.  Part  1  consists  of 
performing  those  activities  required  in  the  planning  of  the  effectiveness  assessment  and 
developing  inputs  to  the  cost-effectiveness  analysis  study  plan.  Part  2  involves  actual 
conduct  of  the  effectiveness  assessment,  to  include  methodology  development  and  the 
analysis  of  the  data.  This  phase  is  closely  coordinated  with  the  cost  analysis  phase, 
because  the  effectiveness  analysis  should  be  planned  and  conducted  in  such  a  way  as  to 
identify,  develop,  and  provide  program  costs  for  use  in  the  cost  analysis  to  obviate 
duplication  of  effort. 

The  cost  analysis  phase  is  conducted  in  parallel  with  and  is  closely  coordinated 
with  the  effectiveness  analysis  phase  to  assure  that  cost  estimates  of  programs  are  valid 
and  reliable.  This  means  that  cost  methodology  must  be  developed  with  full  awareness 
of  the  characteristics  of  the  job  transition  programs  to  be  compared  so  that  these 
analyses  are  relevant  and  the  effectiveness  assessment  effort  can  be  tasked  to  develop 
and  to  provide  appropriate  cost  data  to  the  cost  analysis  efforts.  ‘Furthermore,  the 
objectives  of  the  cost-effectiveness  analysis  shape  the  orientation  of  the  cost  analysis. 


25 


The  cost  analysis  phase  involves  developing  cost  methodology,  determining  cost  data 
requirements,  identifying  cost  data  sources,  obtaining  cost  data,  performing  cost  data 
analysis,  and  determining  the  cost  of  each  job  transition  program  alternative. 

In  the  cost-effectiveness  phase  the  results  of  the  cost  analysis  and  effectiveness 
analysis  are  brought  together.  The  objective  of  this  phase  is  to  integrate  quantitative  and 
qualitative  data  concerning  the  alternatives  and  to  provide  a  solid  analytical  basis  for  the 
decision  maker  to  make  a  decision  concerning  the  alternatives.  This  is  accomplished  by 
deriving  sets  of  quantitative  relationships  describing  relative  costs,  relative  effectiveness, 
relative  worth,  relative  efficiency,  parameter  dominance  relationships,  and  relative 
benefits  of  the  transition  program  alternatives  being  compared.  Analysis  of  these 
relationships  provides  a  basis  for  rank  ordering  the  job  transition  program  alternatives  in 
terms  of  their  relative  effectiveness  in  attaining  the  performance  objectives  and  the  cost 
and  resources  required  to  attain  these  objectives.  These  results  will  be  the  basis  of 
recommendations  presented  to  the  decision  makers. 

The  analysis  of  the  data  from  the  60%  sample  should  be  completed  within  five 
months  after  receipt  of  OMB  questionnaire  clearance.  All  analyses  should  be  completed 
within  seven  months  after  receipt  of  OMB  clearance. 

Preliminary  and  Final  Reports 

A  preliminary  report  based  on  the  results  of  the  analyses  of  the  60%  subsample 
should  be  prepared.  This  report  should  concentrate  on  the  presentation  of  information 
most  relevant  to  answering  the  basic  JAC  evaluation  questions.  However,  the  tentative 
nature  of  the  findings  should  be  stressed.  This  report  should  be  examined  closely  by  the 
project  COR  and  ACAP  monitoring  staff  to  make  sure  that  further  types  of  analyses  and 
data  tables  are  not  warranted  in  their  opinion. 
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A  final  report,  which  would  describe  the  purpose,  methodology,  and  final  results 
of  the  evaluation  as  well  as  any  conclusions  and  recommendations  of  the  authors,  should 
also  be  prepared.  Briefing  charts  highlighting  major  methodological  aspects  and  results 
of  the  evaluation  should  also  be  made.  Sufficient  time  should  be  allowed  for  COR,  peer, 
and  ACAP  review  of  the  final  report  and  briefing  charts. 

The  preliminary  report  should  be  completed  within  six  months  after  receipt  of 
0MB  questionnaire  clearance.  The  final  report  should  be  completed  within  eight  months 
after  receipt  of  OMB  clearance. 
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